最大熵判别主题模型的高效学习算法

doi:10.16451/j.cnki.issn1003-6059.201908007

摘要
图/表
参考文献(29)
相关文章 (1)

全文: PDF (682 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要

现有的有监督主题模型训练算法的时间复杂度一般线性于主题数量,限制了其大规模应用.基于此种情况,文中提出最大熵判别潜在狄利克雷分配(MedLDA)有监督主题模型的高效学习算法.算法为坐标下降算法,训练分类器的迭代次数少于MedLDA已有的蒙特卡洛算法.算法还利用拒绝采样及高效的预处理技术,将训练的时间复杂度从线性于主题数量降至亚线性于主题数量.在多个文本数据集上的对比实验表明,相比原有的蒙特卡洛算法,文中算法在训练速度上有大幅提升.

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	陈键飞
	朱军

关键词 ：有监督主题模型, 坐标下降算法, 吉布斯采样算法, 拒绝采样算法

Abstract：

Time complexity of the existing supervised topic model training algorithms is generally linear to the number of topics and therefore their large-scale application is limited. To solve this problem, an efficient learning algorithm for maximum entropy discrimination of latent Dirichlet allocation(MedLDA) supervised subject model is proposed in this paper. The proposed algorithm is based on coordinate descent, and the number of iterations of training classifiers is less than that of the existing Monte Carlo algorithm for MedLDA. The algorithm also makes use of rejection sampling and efficient preprocessing technique to reduce the time complexity of training from linear to sub-linear with respect to the number of topics. The comparison experiments on multiple text corpora show that the proposed algorithm makes a great improvement in training speed compared with the existing Monte Carlo algorithm.

Key words： Supervised Topic Models Coordinate Descent Gibbs Sampling Rejection Sampling

收稿日期: 2019-05-12

ZTFLH:

TP 181

基金资助:

国家自然科学基金重点国际合作项目(No.61620106010)、北京市自然科学基金重点专题项目(No.L172037)资助

通讯作者: 陈键飞(通讯作者),博士,主要研究方向为大规模机器学习、概率推理、主题模型.E-mail:chenjian14@mails.tsinghua.edu.cn.朱军(通讯作者),博士,教授,主要研究方向为机器学习.E-mail:dcszj@mail.tsinghua.edu.cn.

引用本文:

陈键飞, 朱军. 最大熵判别主题模型的高效学习算法[J]. 模式识别与人工智能, 2019, 32(8): 736-745. CHEN Jianfei, ZHU Jun. Efficient Learning Algorithm for Maximum Entropy Discrimination Topic Models. , 2019, 32(8): 736-745.

链接本文:

http://manu46.magtech.com.cn/Jweb_prai/CN/10.16451/j.cnki.issn1003-6059.201908007 或 http://manu46.magtech.com.cn/Jweb_prai/CN/Y2019/V32/I8/736

[1] BLEI D M, NG A Y, JORDAN M I. Latent Dirichlet Allocation. Journal of Machine Learning Research, 2003, 3: 993-1022.
[2] BLEI D M, LAFFERTY J D. Correlated Topic Models[C/OL]. [2019-04-21]. http://people.ee.duke.edu/~lcarin/Blei2005CTM.pdf.
[3] BLEI D M, LAFFERTY J D. Dynamic Topic Models // Proc of the 23rd International Conference on Machine Learning. Berlin, Germany: Springer, 2006. DOI: 10.1145/1143844.1143859.
[4] CHEN J F, ZHU J, LU J, et al. Scalable Training of Hierarchical Topic Models. Proceedings of the VLDB Endowment, 2018, 11(7): 826-839.
[5] BLEI D M, GRIFFITHS T L, JORDAN M I, et al. Hierarchical Topic Models and the Nested Chinese Restaurant Process[C/OL]. [2019-04-21]. https://people.eecs.berkeley.edu/~jordan/papers/lda-crp.pdf.
[6] MIAO Y S, YU L, BLUNSOM P. Neural Variational Inference for Text Processing // Proc of the 23rd International Conference on Machine Learning. Berlin, Germany: Springer, 2016: 1727-1736.
[7] WEI X, CROFT W B. LDA-Based Document Models for AD-HOC Retrieval // Proc of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM, 2006: 178-185.
[8] WANG C, BLEI D M. Collaborative Topic Modeling for Recommending Scientific Articles // Proc of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, USA: ACM, 2011: 448-456.
[9] LIU S X, WANG X T, LIU J F, et al. TopicPanorama: A Full Picture of Relevant Topics // Proc of the IEEE Conference on Visual Analytics Science and Technology. Washington, USA: IEEE, 2014: 183-192.
[10] WANG Y, ZHAO X M, SUN Z L, et al. Towards Topic Modeling for Big Data[C/OL]. [2019-04-21]. https://arxiv.org/pdf/1405.4402v1.pdf.
[11] DONG Y P, SU H, ZHU J, et al. Improving Interpretability of Deep Neural Networks with Semantic Information // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 4306-4314.
[12] MCAULIFFE J D, BLEI D M. Supervised Topic Models // PLATT J C, KOLLER D, SINGER Y, et al., eds. Advances in Neural Information Processing Systems 20. Cambridge, USA: The MIT Press, 2008: 121-128.
[13] LACOSTE-JULIEN S, SHA F, JORDAN M I. DiscLDA: Discriminative Learning for Dimensionality Reduction and Classification // KOLLER D, SCHUURMANS D, BENGIO Y, et al., eds. Advances in Neural Information Processing Systems 21. Cambridge, USA: The MIT Press, 2009: 897-904.
[14] ZHU J, AHMED A, XING E P. MedLDA: Maximum Margin Supervised Topic Models. Journal of Machine Learning Research, 2012, 13: 2237-2278.
[15] ZHU J, CHEN N, PERKINS H, et al. Gibbs Max-Margin Topic Models with Data Augmentation. Journal of Machine Learning Research, 2014, 15: 1073-1110.
[16] YAO L M, MIMNO D, MCCALLUM A. Efficient Methods for To-pic Model Inference on Streaming Document Collections // Proc of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, USA: ACM, 2009: 937-946.
[17] LI A Q, AHMED A, RAVI S, et al. Reducing the Sampling Complexity of Topic Models // Proc of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, USA: ACM, 2014: 891-900.
[18] YU H F, HSIEH C J, YUN H, et al. A Scalable Asynchronous Distributed Algorithm for Topic Modeling // Proc of the 24th International Conference on World Wide Web. New York, USA: ACM, 2015: 1340-1350.
[19] YUAN J H, GAO F, HO Q R, et al. LightLDA: Big Topic Models on Modest Computer Clusters // Proc of the 24th International Conference on World Wide Web. New York, USA: ACM, 2015: 1351-1361.
[20] CHEN J F, LI K W, ZHU J, et al. WarpLDA: A Cache Efficient O(1) Algorithm for Latent Dirichlet Allocation. Proceedings of the VLDB Endowment, 2016, 9(10): 744-755.
[21] JIANG Q X, ZHU J, SUN M S, et al. Monte Carlo Methods for Maximum Margin Supervised Topic Models // BENGIO S, WALLACH H M, LAROCHELLE H, et al., eds. Advances in Neural Information Processing Systems 25. Cambridge, USA: The MIT Press, 2012: 1592-1600.
[22] ZHU J, CHEN J F, HU W B, et al. Big Learning with Bayesian Methods. National Science Review, 2017, 4(4): 627-651.
[23] MEI S K, ZHU J, ZHU X J. Robust RegBayes: Selectively Incorporating First-Order Logic Domain Knowledge into Bayesian Models // Proc of the 31st International Conference on Machine Learning. Berlin, Germany: Springer, 2014: 253-261.
[24] KOYEJO O, GHOSH J. Constrained Bayesian Inference for Low Rank Multitask Learning // Proc of the 29th Conference on Uncertainty in Artificial Intelligence. Bellevue, USA: AUAI Press, 2013: 341-350.
[25] GRIFFITHS T L, STEYVERS M. Finding Scientific Topics. Proceedings of the National Academy of Sciences of the United States of America, 2004, 101(s1): 5228-5235.
[26] ZHENG X, YU Y L, XING E P. Linear Time Samplers for Supervised Topic Models Using Compositional Proposals // Proc of the 21st ACM SIGKDD International Conference on Knowledge Disco-very and Data Mining. New York, USA: ACM, 2015: 1523-1532.
[27] HSIEH C J, CHANG K W, LIN C J, et al. A Dual Coordinate De-
scent Method for Large-Scale Linear SVM // Proc of the International Conference on Machine Learning. Berlin, Germany: Sprin-ger, 2008: 408-415.
[28] SHALEV-SHWARTZ S, ZHANG T. Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization. Journal of Machine Learning Research, 2013, 14: 567-599.
[29] LEWIS D D, YANG Y M, ROSE T G, et al. RCV1: A New Benchmark Collection for Text Categorization Research. Journal of Machine Learning Research, 2004, 5: 361-397.